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[57] ABSTRACT 

A data transmission mcdiod quickly and reliably transfers 
data (e.g.^ a computer file) from a source to recipients. While 
the frames are being transmitted, negative acknowledgments 
from rec^ients are received by the source. These acknowi- 
edgmeots indicate whidi frames require r^ransmission. 
After all frames have been transmitted out a retransmission 
is performed by the source for only those frames which the 
acknowledgments indicate require retransmission. Addi- 
tional retransmissions can occur. This multi-pass data trans- 
fer technique requires only negative acknowledgements to 
t)e sent by the rec^ents. Features include the ability to set 
the transmission rate and to define multicast groups. Also, it 
is possible to determine the cecity of linlcs of unknown 
oqiacity using a **multicast n^wcrk prc^** feature of the 
invention, and to detennine the frame error rates d known- 
c^ttcity links by utilizing the same feature. A "multicast 
ping** feature iA the invention can be used to determine the 
connectivity between a source and members of a multicast 
group. '•^)eed groups** can be set up after determining link 
capacities, or if they are already known. lA^creby the recq)i* 
ents connected to the source by the fastest links receive all 
of the data while slower-link recipients receive only a 
pMtton of the data, on the first pass. The number 
redfaents ^t^uch can receive the data from the source can l>e 
greatly increased by using a *1iegative acknowledgement 
coUcctirai*' schenoe whereby 'deification points** (jaeferably 
routers) collect individual negative acknowledgements and 
forward them as a unit to the next kveL 
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METH(H>S FOR TRANSMTITING DATA 

CROSS-REFERENCE TO RELATED 
APflJCAnON 

This is a continuatiOD-in-part of U.S. patent applicatioD 
So-. No. 08/375,493 (attOTiey dock^ no. PSM-001) which 
was filed on Jan. 19, 1995, now U.S. Pat No. 5.553.083. 

FIELD OF THE INVENTION 

This invention relates to data transmission, and more 
particularly, to fast and reliable multicast transmissions of 
files from a server to clients. 

BACKGROUND OF THE INVENTION 

Computer networks, such as wide area networks (WANs), 
can provide unicast multicast and broadcast services to 
allow communication between network participants such as 
a server node and one or more client nodes. Multicast frame 
relay is a service used to communicate over a conqniter 
netw<^k. Multicast IP technology is another service used to 
communicate over a computer network. Broadcast frame 
relay is a service used to communicate over a satellite 
network. The term "broadcast" refers to a server node 
sending information to all of the client nodes connected to 
die netw<xk. The term ^^multicasT refers to a server node 
sending information to a subset of all of the client nodes 
connected to the network. Broadcast and multicast are 
netw<»k capabilities which are relatively new over WANs. 

Some information providers desire to deliver information 
electronically by broadcasting or multicasting the inf(»ma- 
tion from a server node at a central location to one or moie 
dicnt nodes at remote customer locations via a coixq)ttter 
network to which the server and the clients arc coupled. 
Because broadcast and multicast network services do not 
provide for acknowledgment <^ the delivered information at 
alL these services can be unrdiaUe. Such unrdiahility 
generally is undesirable and unacceptable to information 
providers. 

A oonanoon protocol suite in use in mmpritrr networks is 
TCPyiE which is the protocol used in die Internet TCP 
stands for Transmission Control Protocol and IP stands for 
Internet RrotocoLTWo file transfer protocols are avail aMc in 
association with TCEWP; (0 File Transfer Protocol (FTP) 
which runs as an application on top of TCP and (ii) Trivial 
Hie TYansfcr Protocol (TFTP) which runs on top of UDP. 
UDP stands ftx User Datagram Protocol Bodi TCP and 
UDP are transport protocols vvhldi arc responsible for 
endr4o-&ad delivery of infonnation across an internetwork. 
Le.. a network of networks. 

Both FTP and TFTP support point-to-point (Lcl. unicast) 
file transfers only. FTP d^ends on TCP for reliable delivciy, 
as TCP is a connection-<Hientcd acknowledged tnuuport 
protocoL TFTP provides its own acknowledgmems for 
reliability, as it nms on top of UDP whidk is a connectionless 
transport service that does not support acknowledgment 

Connection-criented protocols such as TCP require setup 
and tear-down of virtual ctrcnit connections. Because d 
their rdatively hi^ cvcriicad* TCPand similar protocols are 
undesirable in networks with inherently poor connections 
su<A as Cellular Digital Packet Data (CDPD) netwo^. 
CDFD utilizes TCP/IP as the primary protocol suite used in 
the netwock. CDPD wireless netwoiks recammend app^car 
tions operate over UW (the connectionless transport Uyts) 
only, and thus TFTP is the file transfer protocol of choke for 
CDPD. 
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TFTP breaks flies up into packets having 5 12 bytes of data 
each, and it then sends each data packet one at a time. After 
eadi data packet is sent TFTP causes the sending node to 
wait for an acknowledgment from the receiving node(s) 

3 before the sending node is allowed to send the next data 
packet TFTP is described, for ejian^le, in a book by 
Douglas E. Comer (fnXemetworking with TCP/IP, Volume /. 
Principles, Protocols, and Architecture, Second Edition, 
Prentice Hall, 1991, Chapter 23, pages 377-390). 

^0 While acknowledgment is a part of TFTR the acknowl- 
edgment scheme used in TFTP becomes v^ Inefficient as 
network delay becomes significant and/or is dlfTerent for two 
or more of the receiving nodes. Like TFTP, sonae other 
known data transfer mechanisms require packet-by-packet 
acknowledgment and thus these other medianisms also arc 
relatively slow at transfening the entire amount of data. 

SUMMARY OF THE INVENTION 

It is an object of the present invention to fvovide both fast 
^ and reliable transmission of files firom a server to one or 
more clients over a communications link. The file transfer 
preferably is a multicast transmission to clients. In general, 
file transfer according to the invention will not suffer any 
reduction in speed, reliability, or efficiency in die face of link 
^ delay, even if that delay is significant and/<v different for two 
or more of die receiving clients. The invention jsovides an 
ideal mechanism for distnbuting computer software files 
electronically. 

^ The commuttications link, whidi couples the server to the 
clients and allows communication therebetween, can be a 
ccMnputcr networic (e.g.. a LAN, a WAN. the Internet), a 
wireless network (e.g.. a packet celhilar data network such 
as CDPD). some combination of these types of communi- 

35 cation mediums, or some other communication medium 
such as. for example, a satellite network which generally is 
a higb-^>eed, hig^i-delay networic 

In accordance with the invention, the clients send only 
negative acknowlec^ments back to the server as die server 

40 is sending the data files. The communication is continuous. 
That is. the server does not stop sending the data to wait for 
the negative acknowledgments from the clients, but instead 
the server receives the clients* negative adraowiedgments as 
the server is transmitting die data. The clients* negative 

43 acknowledgments indicate to the server which particular 
packets need to be resent Apacket may need to be resent 
t)ecause. for exanqple. It was either not received or received 
in eiTor by one or more of the clients. After the server has 
sent the entire amount of data (e.g.. the entire file) over the 

50 link to the dients. the server performs a second round of 
transmissions in which it only resends the particular packets 
indicated by the clients as requiring retransmission. During 
this second round, clients still only send negative acknowl- 
edgements (i.e.. indications of packets not received at all or 

55 not received correctly). The process can then continue widi 
as many additional rounds of retiansmissiotts as is required 
so that each of the clients correctly receives all of the 
packets. Alternatively, the retransmission rounds can be 
repeated a predetermined number of times, which number 

^ can be modified (Lc the number is configurable). Each 
subsequent round typically involves die transmission of 
fewer packets than the previous round, as only previous 
packets in error are resent 
This schemt quickly and reliabty transfers dau from a 

65 server to one more clients, fi is quick because die server 
is allowed to transfer the entire file without stopping at 
packet boundaries to wait for negative acknowledgments 
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from the dicnts for the packet just sent That is, data transfer 
is not directly tied to negative acknowledgments in that each 
round of data transfer continues regardless of any particular 
dient^s recq>tiQQ pr<^lenis and/cr regardless of any link 
delay Issues (e.g.. a diffcrettce in the time it takes a packet 5 
to travel from the server to a certain client and the time it 
takes a packet to travel frcnn the server to another different 
client). Also, each subsequent round of transmission only 
involves the sending of packets which were not received, or 
received in error, during the previous round and therefore 
the server generally docs not ever need to send the entire file 
more than once. It is reliable because it strives to provide 
each client wi& every packet, and the reception problem of 
any individual client generally does not affect the other 
clients' reception speed and accuracy. 

Data transfer acc<H4ing to the invention does not require 
or expect positive acknowledgements from any of the di- 
ents. A positive acknowledgement is implidt if a negative 
acknowledgement is not received back at the server. 
Moieovex. in accordance with the invention, a plurality oi 20 
negative acknowlcdgnnents preferably are collected and 
sent back to the scrvex as a "multiple selective reject 
negative admowiedgement'* Typically, more than one of 
diese mult^e sdective reject negative acknowledgements 
are sent back to the server during, fox exan^le, the first 25 
round of transmissioas from the server to the clients. One 
nmltiple sdective reject negative acknowledgement can 
r^vesent hundreds of Individual negative acknowledge- 
ments. The use of these collections of negative acknowl- 
edgements can greatly reduce trafScov^ the link and free up 30 
bandwidth on die link for the transfer of data from the server 
to the clients and for odier uses. With the invention, the 
server and the link generally do not get choked witti indi- 
vidual negative admowledgements all condng back at the 
same thne or within a ^ort window of time. This reduction 35 
in the number of individual acknowledgements being sent 
over the link to the server also results in the benefit and 
significant advantage of improved scalability. That is, with 
the use of multiple selective reject negative 
acknowledgements, the number <^dients to which a file can 40 
be sent increases due to the reduced admowledgemeitt 
trafiSc coming back to the server. 

In a preferred embodiment of die invention, ihe entire 
amount of data to be tranrfetred (e.g.« a file) is sq>aratedinto 
a plurality of Mocks, vAmc eadi block includes a plurality 45 
of packed The server ooinplctes a round when it finishes 
transmitting all blocks (eg., the entire file). After a compktc 
block has been transn^tted. tiie clients send thdr negative 
acknowledgments back to the server via a return unicast 
commnnirntions path. Block boundaries trigger the sending 50 
<^ negative acknowledgments by &e clients. As the negative 
acknowledgments are coming into the server from ihe 
clients for block N. the server is transmitting block N4-1 (or 
a subsequent block) out to ihe dients or the server has 
finished transmitting all the tdodES. 5s 

The fdlowing features arc provided according to the 
inventioa. There b the ability to set the transmission rale and 
to define nuiMcast grotips. Also, it is possible to determine 
tite cqfiadty of links (tf unknown ae^dty using a**multicast 
network probe** feature, and to determine the frame error «o 
rates of known-capacity links by utilizing the same feature. 
A '^multicast ping"* feature can t>e used to determine the 
connectivity between a source and nkemfoers of a soulticast 
group. **Speed groups** can be set up after detennining link 
capacities, orif they arc already known,, whereby the recyi- 65 
ents connected to the source by the fastest links receive all 
of the data while slower-link recipients receive only a 
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portion of the data, on the first pass. The number of 
redpients which can receive the data from the source can be 
greatly increased (e.g., by a factor of 1000 or more) by using 
a '"negative acknowledgement collection** scheme whereby 
^q>licatiQn points**, preferably routers, collect individual 
negative acknowledgements and forward diem as a unit to 
the next IcvcL 

It is noted that the terms *packet*. 'datagram*, and *frame* 
are used interchangeably herdn to identify the same thing, 
namdy a unit of data or information which may have a 
source and destination address as part thereof and which is 
sent across the link. 

The f(M:egoing and other objects, aspects, features, and 
advantages of the invention will become more apparent from 
the following descr^on and from the claims. 

BRIEF DESCRIFnON OF THE DRAWINGS 

In the drawings, like reference characters generally refer 
to the same parts throughout the different views. Also, the 
drawings are not necessarily to scale, enophasis instead 
generally being placed upon illustrating the prindi^es of the 
invention. 

FKj. 1 Is a flowdiart of data transmission operations 
according to ttie Invention. 

FIG. 2 is a diagram of a physical configuration ^ch 
allows a swer to cooununicate with one or mote clients. 

FIG. 3 is a diagram showing the location of an embodi- 
ment of the inv^oD in relation to the TCP/IP protocol 
stack. 

FIG. 4 is a diagram of a '*first pass** block and frame 
transmission and acknowledgment process acceding to the 
invention. 

FIG. 5 is a simplified block diagram of a server in which 
at least aportion of the present invention can be embodied. 

FIG. 6 is a diagram a heterogeneous multicast network 
with meinbers of a multicast group connected by different 
capacity links. 

FIG. 7 is a diagram iUnstrating an acknowledgetnent 
coHectiofi feature according to the invention which increases 
sealability and allows millions of rec4E>ients to reodve 
quickly and reliably data from a sender. 

FKj. 8 is a diagram related to congestioo/flow control 
using a variable block size mediod 

FiG. 9 is a diagram related to congestion^aw control 
using a prefezred status request mediod to solidt negative 
ackDOwledgements from dients before block boundaries. 

DBSCRIFTtON 

Referring to FIGS. 1 and X in accordance with the 
invention, quick and reliable dau transmission from a 
source or server 20 to one or more recqifents or rcodvers or 

clients 22^, 22, 22^^ over a communicatiotts link 24 

comprises (stq> 10) transmitting the data (e.g,. a file), which 
is in the fozm of aphirality firames, over the link 24 to one 
or roorc of the recq>ients 22 until the entire file (i.e«, all of 
the plurality of frames) have been transmitted over the link 
24. As file frames are being transmitted, frame negative 
acknowledgments from cue or more of the redpients 22 are 
received via die link 24 (step 10)l If, after the entire file has 
beta transmitted over the link 24. the negative acknowledg- 
ments indiratr that certain frames need to be retransmitted 
over the link 24 (step 12), only those certain frames are 
retransmitted (st^ 14). As those certain frames are being 
retransmitted over the link 24, frame negative acknowledg- 
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ments from one or more of the recipients 22 are received via in other protocol stacks such as IPX in the NetWare SPX/ 

the link 24 (step 14), This process is then repeated as many IPX protocol suite, UDP stands for User Datagram Protocol 

times as necessary until no more frames need to be and it is the TCP/IP standard protocol that allows an appli- 

rctransmitted, as indicated by steps 12, 14, and 16. ia step cation program on one computer to said a datagram to an 
16, the servCT 20 dctennines whether •'done" messages have 3 (plication foogram on another computer. IJDP uses flic 

been received at the saver 20 by aU of the recq)ients 22. If fct^rt Protocol (IP) to da^ 

aredpientis-done-Umeansttkredpienthasreceivedall ^Jl?^ ^^£f°*?."? ^ '^^^'^f ^Th^ h"^^ 

of theframesandhassenttothe serva^a^oiic-mcssage Protocol port nujriber which aUows die sender of the dto- 

to so indicate «I>one'' tecj>^nts «,ndnue to send"S^ ^^n^^S^nT^^m^'^ 

m^sages to the server until they ^^^^ W/^t also ty^icSy indude a checksiii f <?the data being sent 

hsf whidi the '^^rfff^i'^.'J''^ m Antral, data tiansmission according to the invention 

rccipients(ix.,thosehstedmthe donehsntostopse^^ includes four aspects: IDLE, ANNOUNCE/ 

**donc" messages to the server. After a iwcdetenmned period REGISTRAnON, TRANSFER, and COMPLEnON. In flic 

of time or after a predetermined event, the server 20 sends jjj^ ^ activity. When a c<^cction of data 
a status request to aU unresponsive rec^ients 22. Le., u (e.g., a file) is selected for transmission by the server 20, the 

recqxients from which it has not received a "done** message ANNOUNCE/RBGlSTRAnON phase is entered. During 

(step 18). The inidal transfio- of the entire file and each of the any of the four phases, all files are available to an opeiator 

subsequent transmissions of error frames are generally at the server 20. 

referred to herein as a •*round** or **pa»s". ANNOUNCE/REGISTRAnON 

In the first pass, the server 20 preferably multicasts the file » In this i*ase{stq> 8 in FIG. 1), the server ANNOUNCE 

to a subset of all of the dicnts 22. At least two of the clients to ^ cUcnts that a file is about to be transfcired and 

22 typicaUy have a different seivcr-to-dicnt frame transmis- provides the parameters associated wi& the transfer of flie 

sion dday associated therewith. Data transmission accord- The maximum duration of this |Aase is esqwessed u 

ing to the invention is unaffected by such delay differences minotcs. ai^i it is configurable. An ANNOUNCE message is 
even if significant and even if every cKent 22 has a diffCTent 23 used to set iq> multicast grwips, and Class D addresses are 

delay assodated therewith. in the assignment of multicast groups. 

. . ^ ^ / f Avr * Clients are obliged to register with the server that they 

«,^fil^T^ a compute nclwo* (eg. a LAN. a reorived an ANNOLNCE^sage. When a cKent sees tte 

WAN. 4e imtroct). ^j^ssj^^s^^^c^ data j^q^j^ce n«sagc Ihe3^.aifi« that it is associ- 

nctw»kXs™«combu«U.onof«^ atcd v4b the g«HvU«=»tified in flic message. It is in^licit 

ni«taoo medmms or s«ne oa«c^^ nx^Tbdng aMe lo proc^tfTANNO^CB 

such as. for example, a satellite network which typically are , ^ . ^ ^ to . . 

* u 35 they sec their address in a registered dlent list in a subsc- 

one or more of the clients 22. ANNOUNCE packet The REGKIRATEON packet 

The server 20 and the clients 22 can be cwnputcfs. such act, as a positive acknowledgment to the sexvcr about the 

as PCs or workstations, running any one a variety of ^^^j.^ participation. Once the sjivcr receives the ciicnt^s 

operating systems including DOS. Referring to FIG. 5, the REGmRAnON packet the server adds the dient to the 

scrvcr20.regardlessofwhattypeotfcoii5Hitcritis,typically ^ diait list in the next bm>adcast of the ANNOUNCE packet 

indudesacentralproccss<r50,amainmcmofyunitS2for The client list is maintained by the server When the cUcnt 

storing programs aad/or data, an input^ou^t controller 54 receives an ANNOUNCE padcet with the ciicnfs ID in the 
a network interface 56, one or more input devices 58 sucA ^ r^istratioo Ux the client is coii9>lcte. When aU 

as a keyboard and a mouse, a display device 60, a fixed « expected lecdvcts have responded to the ANNOUNCE 

hard disk drive unit 62, a floppy disk drive unit 64, a tape ^ niessage or the ANNOUNCE timeout has expired, which- 

dtive unit 66, and a data bus 68 coupling these coc^nents ^ ^ transmission of the file wiU begin, 
to aUow communication therrt)ctwcciL Eadi of the dlent registration indicates that the dient can participate in 

conq>utcxs 22 generally indudes all or some <rf the cOTpo- the group, as it has Ac resources to handle the file about to 

Dents induded in the server 20 of FIG. 5. ^ prevent unwanted particqxatiott, encryption key 

In some embodiments, one or more con^uter programs 30 exchange can take place at group setup. Once file transfer 

define the operational capabilities of the server 20 and the begins, ANNOUNCE packets cease to be sent and Ae 

clients 22. The programs can be loaded into the server 20 ANNOUNCE phase is over (step 9 in FIG. 1). 
and the clients 22 via the hard drive 6Z the floppy drive 64% the characteristics of the file transmission are trans^ 

and/oir the t^ drive 66. Attcmativdy. the programs can mitted in the ANNOUNCE packet On recdving this 

reside in a permanent memoiypQitioD(e.g.. a ROM diQ)) of 33 ANNOUNCE messi^, the dient req^xwds with a unicast 

fte nnain memory 52.1* soincolhercinbodimeiits, the server datagram to the server. The response indicates whether or 

20 and/or the clients 22 can indude spedally-designed, not the receiver has the fadlities to recdve the file. It also 

dedicated, hard-wired dectronic circuits whidi perfonn all indicates, in the case of an aborted transmission, whether the 

fiinctioos described herein without the need for instructions dient has enough context to resume the transmission (a 

from computer piogranks. The invention can be used, for ^ *tetarr as indicated in FIG. 1). The duration of the 

example, to load quickly and reliabty new levision Icvds of announce period In some, instances should allow for an 

the client software electronically from the server onto one or cperator at the server site to initiate a call to the client site 

more of flic clients. indicating that tfac oooqMilcr is dtfacr not available or does 

Refening to FKj. 3, the inventkm preferably operates at not have the facilities for the transfer. At the dient site, the 

flieq)]^icatk)n layer 30 of the TCP/IP protocol stadc 32 on 65 conections could be made either manually or, if so 

top of UDP. The invention also could operate at flic appli- configured, under remote control from the server to fm up 

cation layer above the connectionless transport layer present resource so it can partidpate in the transfer. 
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At any point in time throughout the transmission, the 
dicnt may respond to this packet indicating that it ^>orted 
the transmissioD from its end indicating the reason in ttie 
message. If a transfer is broken ofif before conqpletion, the 
invention is able to resume later without resending parts of 
the file already sent successfully ('"restaxT in FIG. 1). This 
is an especially in^xxtant and useful feature when sending 
very la^e files. To achieve this feature, a dient does not 
discard a partially received file. Instead, the clients store 
paitially received files. If there arc problems that prevent all 
clients (e.g.. all clients in a multicast group) from receiving 
the entire file when it is first being sent (e.g., the link is 
terminated fcr some reason during file transmission), the 
transmission can be restarted later to complete the transfer 
During a restart the server queries all clients for alist of data 
frames that were missed, and then the server begins the 
conviction of the transfer by sending only those firames. 
Thus, in FIG. 1. for a restart step 10 involves a transmission 
that starts first with the frames that were noissed (Le., Nak'd) 
during the initial aborted transmission, instead of starting 
with the first frame of the first block of the file as would 
happen in an unaboited nc»mal start <^ a transfer. 
TRANSFER 

Up<Hi eatedng the data transfer phase, a transmission log 
is maintained at the server This log is always on, and it 
keeps track of all events. Each of tiie clients also maintains 
a transmission log. The log maintained at each cf the clients 
is mentioned hereinafter under the tX)MHJmON** head- 
ing. 

As files having 2 gigabytes of data or more can be 
transferred, holding die entire file in memory at the server 
for the extent of the transfer generally is unrealistic. The 
number of clients whi<^ are to receive the file can be 1000 
or more, and thus halting transmission to wait tot acknowl- 
edgments from each of them before continuing on to the next 
block transfer is unaccq;»tablc. 

The server logically breaks each file to be transferred into 
blocks of frames, and eadi block typtcaUy includes a plu- 
rality of frames and possibly thousands of frames. Refexring 
to FIG. 4, in one exan^le, the server 20 has broken a file into 
four blocks, nauMly, Block 1, Blodc 2, Bloc^ 3. and. Block 
4 .wherein each block includes one or more frames. Each 
blockrcpresents a unit that will be negatively acknowledged 
(only, no positive acknowledgements) by every client par- 
tic^>ating in a transfer when the client detennines that a 
block has been sent by fike server The client detects tbis by 
a diange in Mock number in data packets received, because 
each frame sent indicates Its block number and its frame 
number within that Uock. Breaking die file into blocks 
provides at least two advantages: (i) decreasing the number 
cf negative acknowledgments required; and (ii) reducing the 
memory re^iirements in iSat server for determtning next file 
pass transfer blocks. 

Data transfers are not directly tied to the negative 
acknowledgments. Transfer continues regardless missed 
negative acknowledgments or previously missed data pack- 
ets by aoy individual dicnt This allows simplirity of design 
and ensures tiiat individual client problems provide Tmimnfll 
impact on the group as a whole. Note also that clients are 
responsible for sending block negative acknowledgments 
based on what they hear from the server. 

Refening to FIG. 4. the server starts tike transfer by 
sending the first frame of tiie first block (ic iht first firame 
of Block^. The server sends the frames at a rate that is 
configurable. This represents the basic transfer rate that may 
be throttled back (Le., decreased) based on performance. 
Hie server continues sending the frames tiie file until the 
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complete file has been sent once into the network (Le., until 
Blod^i through Blods^ are sent). This is defined as the first 
pass or first round, and it takes an amount of time icprc- 
sented in FIG. 4 as "B4'*. Some clients may receive the 

5 conq>lete file (Le., all four blocks) correcdy after the first 
pass, in which case they have finished receiving the file. 
Clients receiving one or more frames in cuor, or not 
receiving one or more frames at all. require the resending of 
certain *t>icces** of the file (Le.. the crroneousiy-rcccivcd or 

10 missed firames) in subsequent passes or rounds. Each sub- 
sequent pass or round requires the transmission of fewer 
frames because only frames negatively acknowledged (Le., 
frames not received or received in error) in the previous 
round get retransmitted in the subsequent round. 

IS A maximum pass count or a maximum time to complete 
can be a configurable parameter. There may be clients that 
have not received all of the file ooircctiy by the time of a 
maximum pass or a maxinnim time duration. These clients 
arc identified by the server, and the server can take further 

20 action to get these clients the rest of the information via. for 
example, a unicast file transfer process. In the preferred 
embodiment the clients send '*done messages'* indicating 
theyVe received the whole file and the server sends "done 
lists** indicating clients said to be ^done.** tf. after a piede- 

25 termioed event (e.g., a predetermined amount of time), the 
server does not receive a ''done message** firom certain 
clients and all NAKs have been serviced, the server sends to 
those clients a status request message and sends any missing 
frames to clients needing mere data. Any client that still is 

30 unresponsive can be sent the file in, tot exan^e. a unicast 
transfer to tiiat dient firom the server at a later time. 

As the server passes block boundaries (Le. .6^.62,63 and 
B^ in FIG. 4), the individual clients preferably send "mul- 
tifit selective reject negative** acknowledgments (^"Nak^ 

35 for each block. These admowledgments from tiie clients for 
each block are received at the server scnnetime after the 
tKHmdary of that block is passed. Positive acknowledge- 
nkents are implicit A mult^le selective reject negative 
acknowledgment for a particular block means tiiat one or 

40 multiple frames in that particular block were received in 
em^. or were not received at all by those clients indicating 
that the netwoik did not deliver them for some reason. Thus, 
acknowledgments sent to the server indicate -wbith frames 
were received in cnror or not received. 

45 On subsequent passes (Lc after the first pass shown in 
FIG. 4). dients only respond with negative acknowledg- 
ments for blocks ^ain not received cocrectiy. Since the 
server sends pieces (frames) of the file needed by various 
clients to all clients in subsequent passes, many of tiie clients 

50 will have already received it coiroctiy on the first pass and 
thus will ignore U. 

In general all infofmation returning back to the server 
firom the clients may be transmitted on a return path which 
is separate from the path(s) which the server uses to transfer 

55 the frames to the clients. However, for the purposes of this 
description, the communicatk»ns link 24 (FIG. 2), or other 
path which allows the server and the clients to communicate, 
should generally be taken to mean botii the server-to-client 
link and the return client-to-server link. 

€0 The server maintains various information atxwt the trans- 
fer and the partidpants In the transfer. In the preferred 
embodinoent, this information is maintained by the server in 
the form of data structures or Hsts. The server maintains and 
uses this information to record and determine the status of 

65 the file transfer. 

The server also maintains a frame data structure whidi 
indicates all selective rejects on individual frames from all 
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dients. If mult^lc dients missed the same firame, the frame 
data structure would indicate CHily that the frame was 
missed. That Is, the frrame data structure is not maintained on 
a client-by-client basis by the saver. It generally is undo- 
siraUe fcsr the server to maintain a detailed list of missed 
frames on a per client basis because such a scheme would 
use an inordinate amount of memory, panicularly when a 
laige number (e.g.. 1000 or more) of clients arc involved in 
die multicast. For example, it might be that one or m<H« of 
the dieots either did not receive or received in error frames 
twenty and twenty-five of Blocks, fnant one Blocks* 
oeitain frames cf Blodc^. etc. If tiie frame status n^ntaincd 
by the server indicates that a particular frame of a particular 
block needs to be retransmitted, it will be tnie that at least 
one of the clients has not acknowledged successful comple- 
tion of that particular block. After the server has sent the 
entire file once, the server would then pass through the fr^me 
status infonnation and resend only the times listed therein. 
Uns would continue, pass after pass, until all clients had sent 
"done messages** and the franac status list is empty (or the 
maximum number of rounds, or TnaTimum time, had been 
reached). 

Note that for any given pass, if any negative acknowl- 
edgment does not get bade to the server, the client will send 
bock to ttie server the same reject and retransmission request 
messages during the next pass fay the server. This means that 
if a certain client is not being heard fay the server, that dient 
will have to partidpatc longer but that dient will not 
appreciable impact the rest of the receiving dients. 

Another piece of information stored at die server is 
statistics on die multicast group. When a transmissioo Is 
conqileted, summaiy inf onn3ti<m is provided on the trans- 
mission that can aid an operator in detcnnining system 
perfonnance prc*lems and/or the performance problems of 
a particular client 
Mult^e P^isses Through the 1%: 

Once the file has been cooqiletcly processed once (Le.. 
after die first pass or round), the tranamssion process 
according to the invention will increment a pass counter and 
then scan the frame status list in ttie sewer for the first block 
in which there was an error. Upon finding this first-enrar 
block, the server will resend the missed packets in that 
block. Negative ackoowlcdgmeDts for these missed packets 
wilL as described previously, he generated by the dienb 
when Ihcy iletect an error in a block. This is consistent with 
die first pass. All selective reject negative acknowledgments 
are indications <^ state and art therefore not specific to a 
pass though they may change with eadi pass. In a preferred 
embodiment, the nwlttple sdective reject negative acknowl- 
edgments are in the form of bitmips where the entire word 
represents a block and each tiit in that word rqHescnts a 
different one of the frames which make up that block. 
lYansmission Abort: 

If. during the transmission, a fault is encountered which 
cannot be rectified, or if the operat<»' manually aborts, a 
transmission abort sequence will be initiated. Ihis sequence 
entails the rq)eated transmission of an Abort message for a 
oeitain interval (e.g.« for an interval which is specified in a 
transmissions file). The receivers acknowledge the abort 
message aixl can take action to. for cxaixq;>lc. either save die 
context for a potential rcsutnptioD (Le., restart) of die 
transfer or reinitialize the context to prqpare for another 
transmission. There is a facility which allows the user to 
initiate a transmissioa abort A reason code can l>e set to 
eidier suspend or initialize. Id the f onner case, die trans- 
mission may be resumed or restarted at a later time, and, in 
the latter case, the dients will be requested to reinitialize 
their contexts. 
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COMPLEnON 

The server detects conqiletion of individual clients by 
recdving a ''done message** fitun a client Ihe client knows 
it*s done as soon as it has all bkxrks of the file, but the client 

s must continue to send "done messages^ until Uie server 
confirms con^etion. The server confirms that a dient is 
"done** by placing that client's address in a "done list" and 
sending die list out to die cUents. When a dient sees its 
address listed in die ^done list,*" it knows it has completed 

10 the transfer. The dient wHI then update its transmission log 
to indicate that die transfer has been successfully completed. 

An ability to abort a transfer from die server or client is 
induded. An abort packet provides the server and cUent with 
the ability to abort prematurely a transfer. If the client sends 

15 an abort, die server removes the client from die group. If the 
server aborts, the transfer can be restarted widioift sending 
die fun file on the first pass. 
Status Requests: 
If, after die first pass, die server has not recdved eidier a 

20 DONE <»- NAK from a dient a query is sent directed to 
those clients whose status is not known. The responses are 
in the form of a standard response message. They will 
indude a bitmi^ describing die errors if there are errors to 
report. 

25 Congestion/Flow Control: 

As large ''internets'* become nudticast enabled, it will 
become more common to find multicast groups that desire 
information to have different transmission links to the mem- 
bers of that group. Ihese different links may have different 

30 ci^Mcides. ^K^iichroay be grcady divergent from each other. 
Fdr example, one member of the group may have a link 
c^suity of over 1 Mbps while another may only have 56 
Kbps. In general, knowledge of these link capadties will not 
be known by die sender (eg., die server) of the transmission. 

35 Ihus, it is desirable to be abie to determine the link 
capacides on the fly, and provide a flow control mrdianism 
to prevent overload/congestion of the network ^i^e at the 
same time not inhibiting the efficiency of the data transfer 
protocol 

40 The data transfer protocol described herein indudes the 
concept of blocks, each one <^ which can contain hundreds 
or thousands of frames. Qieitts (recqnents) are obliged to 
send a multqile selective reject NAK at block boundaries if 
any frames are missing or in error in that block. For flow 

45 control purposes, it is desirable to gain knowledge of 
missed/erroneous (Le., dropped) frames as soon as pracdcaU 
so flow control dmsions can be made. Changing or variable 
block sizes is a way to accomfdish this widi the data transfer 
protocol of die invention, and this invdves starting with a 

50 relatively small block and increasing blodc size during file 
transfer to fce^ current scalability by reducing dient 
acknowledgements. Another, preferred way to accomplish 
this widi the data transfer protocol (^die invention is to keep 
the block sizes all the same (homogeneous block sizes) but 

55 let die server send out status requests before a Mock bound- 
ary occurs so that clients rc^nd with NAKs l>efare die 
block boundary. This latter technique is the most flexible, as 
NAKs may be solicited at any time, as opposed to just at 
block bouiKiaries which is the case with the fonner tedi- 

60 nique. With dther of these two techniques. NAKs are 
solidted eady in the transfer. 

In the *Wari8ble block size** method (the first technique 
mentioned in the preceding paragr^X die first block may 
be reUtively smalL c.g.. 100 firames. Subsequent hk>dcs 

65 increase by a factor of two each time. Block sizes are 
inaeasingly doubled until the maximum block size is 
readied or the file reaches its end. 
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lb the "status request** method (the second technique dropped by 15%, or a higher or slightly higher percentage. 

mcntioDed previously), the server solicits NAK responses at to accommodate client B. 

points where it desires, and these points are not at block The timing of the van^le block size method is given in 

boundaries. With this piefcired embodiment to congestion FIG. 8. As soon as infonnation at a client indicates its frame 

or flow control, status requests are sent at increasingly 5 drops exceed the Group Thresholds that client must take erne 

longer intervals. of the three above-listed alternative actions so that the group 

"^th both methods, transmission rate or transfer rate is set transmission is not adversely afifected. The adjustment of the 

as described herein. However, rather than a fixed transfer group* s transfer rate is perfonned alter the second block has 

rate, the settable rate represents an upper bound for die been sent, starting with the beginning of block 3. lYansfer 

transfer rate. After the first block (with the variable block lO rate changes arc inqilcmcotedat block boundaries to provide 

size method) or when a status request is received (with the accurate data on a block basis from the block NAKs. The file 

status request method), NAKs arc sent to the sover by transfer then proceeds with the transfer of block 3. which is 

clients that have dropped frames, and this is an indication of set to be twice as large as block 2, just as block 2 is twice 

congestion by those clients to the server. as large as block L This is followed by block 4. whidi is 

If there are NAKs, the fact that they are directly related to is twice as large as blo(± 3. and so on until the maximum block 

the instantaneous c^dty oi the particular link can be used size is reached cr the file reaches its end. whichever occurs 

to determine link capacity for all of the links diat show first However, if NAKs from the group after block 3 

congestion based on the foUowii^ equation: indicate the worst client exceeds a Rate Hircshold parameter 

(configurable), then the rate is further adjusted fc^ block 5 

20 transniission. The Rate Threshold is the min'm"!" frame 

((«&aiQM 8ent-#fraines NAK'dy#framefl 8eor)*tiaiisfex raie=iink percentage for which transfer rate adjustments for the 

group are performed. For exan^le* a maximum fi^une drop 

In the hctoogeneous mutticast network of FIG. 6, link P^^^ of 1% tem Ae clients would not warrant an 

speeds range from64 Kbps to 1024 Kbps, a large difference adjustmait so the RateTTircshoW would typically be set to 

in Unkc^dty. Assuming no other traffic. If the transfer rate " a mmrt>er above 1%. ^ ^ ^ ^ 

w«e set for 150 Kbps, the blodc NAK for Mode 1 from ^ the status request method, deblocks area umform size 

dient A On the variable block dze method) would indicate requests arc seat by the server to request NAKs 

about 58 frames dropped for die first blodL Using the above >>ef^ Wock boundaries are readied. ReferringtoHG. 9, an 

equation, the instantaneous link q)ecd is calculated to be 63 ^<^^ ^ Jllf^!?"'^ descrtbed f or v^g 

Kbps. The block NAK from dient B for block! On the ^ siKblodc method is depicted except now (in 

variable Mode size method) would indicate about 15 frames ^ ^« ^Jf^^Sf^ ^ ^^f""" ^"^l 

droM>ed for the first block, Again using the equation, the '^^^ is sent after 100 frames of the transfer, the second 

instantaneous link speed fee the linkto B is calculated to be after 200 mwe frames are sent etc. Client NAKs are sent 

127J Kbps.Withofeertiafficprcsenttiienumberofframes tecfc to the sewer ai exactly Ite 

dropped would be higher resulting in a srnallcr calculated Wock «ze method Howcv«. th« 

link speed. with the status request method that status requests are sent 

AGroupThresholdparamctcrmay be setby the usd^TTic ^ ^ * 

Group Threshold is die Emit expressed in percent of bouiidap to rcc«ve NAKs as is the case with ^ 

drcpped frames, by a particular client that is allowed for . . a. . 

continuing participation in the nadticastgioiH). If the Groim ^ With athcr the variable block size method or the statos 

Threshold is set to 25%, it means that any dients in the request incthod, genoaUy is^not desi^^ 

group that have a frame drop percentage higher than 25% members and leave them hangmg. Ddeted group 

win need to take action so that the rest of the group is not mcnibcis <^ be colledcd into ^ 

ad vorsdy affected In the example of HG. 6, dient A with ^"^^ «^ ^^wcr transfer rate may be deter- 

58% of the frames dropped would need to take action. mi^cd by tte calciilatK>n on 1^ 

Clients wiU have enough information to make tiiat dedsion who leave the group. This group can then be set up 

because the transfer rate and Group Hircshoid parameters ^^"^ rate and a new transfer can be 

are transmitted to clients in die Annouace message. Otents ip'***™ , ^. ^ 

whididetectthattheirframedropratcexceedsthethreshoW BoA Ac variable block size and the status request mcth- 

may take one of the following actions: * ods of Ac flow contrd process can be made automatic. 

1. l^ve theGioupj^ scrvotobeput ^J^t can be in two forms: appUcation layer (AL) 

into a lower speed group, with the ?«ip speed spca- nudlicast where the netwodc stiU ddivcrs data to the entire 

fied based on the measurement made at that dient; broadcast group, and multicast IP where the network routes 

2. Leave die Group without requesting further dcUvciy, 55 traffic based on muWcast routers and Internet spedflcation 
meaning that this dient misses this transmission; and Rpc 1112 is inqteneated in die clients. 

3. Suppsess NAKs until a Status Request message is la both cases, multicast groups arc set up under initiation 
received from tfie Server, allowing the rest of the group of the seivcr. The server sends notifications on a unicast 
to finish without being held up by excessive retrans- basts to dicnts to inform them of membership in a particular 
missions from a high frame loss dient (the transfier rate 60 multirast group. These multicast groups can be set up and 
for retransmissions to this set of dients could be lower diiqrwntlpd rapidly, allowing for a dynamic configuration of 
to reflect their lower capacities). multicast groups. F<^ example, a multicast group could be 

In die example of FIG. 6. the nesit h^est percentage of set up to be only in place for die transmission of a particular 

frame drops comes frcmi diem B at 15% whidiis underthe file, after which dme the group was dismantled 

(jroup Threshold This nuniber represents a factor to which 6S With AL multicast the netwoik still delivers traffic on a 

the whole group can adjust without undue performance broadcast basis, but dicnts not in the group discard die data 

degradation. The server's transfer rate far the group is then not destined for it When the group is set up. security keys 
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may also be disseminated so that dients outside the group a packet could contain would be 256(bytcs4[>ackct) * 8 

cannot read Ac data even if it h^ncd that the data was not (bits/Byte>=204S (hits/i>ackct) whidi means that toe largest 

discarded at that node (note that this could also be deployed allowable block size would be a block having 204«packct5. 

with multicast IP). Alsa with AL multicast the IP address Although /f^^^f interfaced to an 

remains a global or network^ broadcast address. As 5 phernc^ at 10 M^s, WAN i^nfaj*;;^ often o^ mt|^ 

with hroad^ this addi^ss becomes mapped toabroadcast *t ^ 

address in the r^^' "^'/f '"""S " SSj^StS' ead. experience resource problems 

address. A mufticast header is selected te the group and ^^^J^^.^^ ^ ^ , tranSon. Receiving Jiodcs arc 

becOTies the group diffcrentialor. ^^^^ resources i»ior to a transmission and 

Wth multicast IP. the netwak is a router network where lo ^^^^^^^^ J ^avc the fecilitics to receive the data. If 

the routers suppcat Class D multicast IP addresses and should either rddtiaUze space which is 

multicast routing. The clients support RFC 1112, "Host dedicated for the transmission or should indicate that they 

Extensions for IPMulticasting". RFC 1U2 provides fwhost cannot participate in the transmission and ooirective mea- 

notification of their presence to the nearest multicast router ^ undotaloen through different channels. A facility 

for the purpose of update of router tables. 13 could be provided where the sewer can force the availability 

A functional desor^on of the above-described invention of disk ^>ace roxK^ly to allow the transfer of the file to take 

is prcyvided below. place. 

Referring back to FIG. 2, which generally can represent The receivers 22 must also be aware of what tb^ are 

any broadcast or multicast IP router-based netw(»k« a pur- listening for. When a datagram is received on a dedicated 

pose of the invention is to enable the simultaneous trans- 20 channel, the node 22 must deteiminc if it is being addressed, 

mission of small or large data files (e.g., files up to 2 An issue can arise whnn this application is being used by 

gigabytes or mwe in size) by a server 20 to up to 5000 or more than one transmission server 20. There must be a way 

more receiving nodes 22 over a wide area network (WAN) of guaranteeing diat a receiving node 22 is participating in 

connection 24. The invention also is able to work ovw local exactly one transmission at a given time. By dedicating a 

area networks and other types of communications links, as 25 UDP port to a server 20 and also relating an encryption key 

described |»cviously. The transmission medium 24 can be to mat saver, it is ensured that a recdving node employing 

any type which supports the TCP/IP protocol stack in tiie a promiscuous mode tap on the networic 24 will not have tfie 

prcfoxed embodiment Odicr protocol stacks could also ability to be able to interpret the transmitted data, 

serve as the communicatioDS environment fcw^ the invention. Some reference informaticm is maintained on the trans- 

Multicastcanbesapp<»tedintwoways:ALrmilticastand 30 misdon server 20. Ttec prcftraibly is a list <^ all tiic 

mnitifatf IP, as mcntioncd previously. potential receiving nodes in the network. Enough reference 

Files to be transferred to the clients can be loaded onto the information preferably is avaflaWe to allow the information 

server 2© via t^(e.g^ the tape drive 66 of FIG. 5) or, if die provider to manage tficcUents in die case of service faUures, 

files are smaU enough by floppy (e.g^ the floppy drive oi problems, etc. There preferably will be a transmission daia- 

FIG. 5). Also, files to be transfened can be loaded onto die 35 base where an encrypted ccmpressed data file is maintaiijed 

server 2* via FTP (File TVansfcx Ptotocol), or some oflier ready for transmission. The transmission database contains 

unicast transfer mechanism, from the sourceof the file over the prepared data along witii descr4>tive inf(smation of up 

a LAN ox other network, for cxanqple. The files generally to, for eawn^ilc. 70 bytes identifying the content of the files, 

can be in any fotmaL The data fie is then read in firom die Each transmission preferably has a completion sUtus 

tape OT floppy into a file system of the transmission server 40 indicator record and a log of all errors encountered during 

20. Note that &e server 20 must have sufSctent space the transmission. There jscferably also Is an event file with 

available to read in an uncrai^xressed copy off the data file. a list all the nodes for which the transmission failed, who 

F6r both services, the daU file also can be encrypted so that to call, and why it failed. 

noneligible receivers cannot receive and use the data file. At any point in time during the transmission, an operator 

Each transmission file prefcralHy is uniquely identified. 45 is aWc to interrogate the status of die transmission as it 

Ihere preferably is an indicaticm as to its content and time spfi^ to die server 20 and each of (he receiving nodes 22. 

generation. The input files to the process can be over 2 Alerts are generated if there are problems coixmmtticating to 

gigabytes in size, and the system can also handle files much certain clients or oftcr problems. If any intervention is 

larger dian 2 gigabytes. indicated, tt» operator is allowed to initiate the ccaicctive 

The file can dien be stored on the server 20 and prepared 50 action, 

far transmission. Data firom previous transmissions will For ongoing m a irtfffn a nc e and naanagement of die service, 

need to t>e readily avaih^ on the server 20 for some period die operator is enabled to maintain die list of receivers, 

of time In case tiiey need to be rctrannnitted. A medianism transmission groups, transmission file descriptors, transmis- 

fbr accessing the data is provided such diat the data can be slon param^ers, and transmissions database. A background 

readily queued-up for retransmission. 55 process wiUinaintain die environnicnt and botti a^ data and 

Rjr^ciency. the file is transmitted in Mocks. The size of delete it according to housekeeping parameters, if enabled 

a blockis derived firom the largest packet (or block size can by an alerted operator. 

be selected by die user) \(4iidi can be transficrred over die Data transmission according to the invention has been 

€ommunicaiionspadi24.Itsdcrivatiooisbascdoodiefact described above. Ftolher aspects of the invention arc 

dial die dients wiU need to indicate to die server wbkt of <o described hereinafter. These finttier aspects indude: SET- 

die padats in a Mock they failed to receive. One way, and TABLE TOANSMISSION RATE; MUUICAST CHIOUPS; 

genffally die sinmlest way. to do this is to send a Mtmap MUITICAST PING; MUUnCACT NETWORK KOBE; 

hidicating by a Mt setting pooticmally which packets were SPEED GROUPS; and NEGATIVE ACKNOWLHXJE^ 

not received. The size of ttie btock ttierefore is approou- MENT OCXLECIION. 

matdy the number of packets which can be acknowledged as SETTABLE TRANSMISSION RATE 

in a Mtmap wMch itself can be contained in a packet For As mentioned previously, it is possiWe to set die data 

exan^le. if die packet size were 256 bytes, dien the most bits transmission rate. The example given previously iUustratcd 
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when a settable rate is useful. In that example, die receiving e.g.. at the server. A multicast group is specified when a 

nodes 22 are interfaced to an Ethernet LAN haying an sending station wants to transmit a file. The group is 

available bandwidth of 10 Mbps and the WAN links con- identified by a list of client IP addresses, one address for 

^^^^^^}^ cadi dicnt in Ihe multicast group, 

than 10 Mbps. In such a case, die data transmission rate < ^^^^^ j™„»,s^ 

would be siT according to the invention, to match, f or ' ?^ are twoj^ons for multicast groups: dymomc and 

example, the speed of the slowest WAN link. "^^^^ 8^P^- ^^en the transfer is 

In accwdance with the invention, for any given file c<Mnpl«c. group dissolves. Dynamic multicast groups 

transfer session, the data transmission rate can be set ahead are formed ^ the ANNOUNCE nicssages usin^ 

of time. More specificaUy, the maximum bit rate at which ,o ^"^^ ^fl ^ ^f^^'^ ^ \? '^"^^^'^ 

data is transmitted during die session is scttaWc. In a ^"^^^ ^ muUicast groups, aU of the members of 

Jli ..Tr *V. ™ »cu«oic. IB a die group rcmab mcmbcrs of the gfoup when the transfer is 

preferred embodiment, it is set by setting a parameter to an _ Jr. / uj . ^ K ^ T " 

. »u * * . . uit uj. complete. Static multicast groups arc formed by the server 

integer value that renresents the bit rate in kilobits t>er ^, ^ . ™ 

^ 17 uic u«r«c m pci ^ umcast basis and/or by using a common Class D 

second (Kbps). For cxan^le, if dus rate parametar has die ^^^^ to set im configurations, 

value 56, it corresponds to a maximum bit rate of 56 Kbps. " MUUICAST FING 

The rate parameter can be set to any value diat coiresponds The "jrfng" utility in TCP/IP is very useftil in determining 

to the avaUable bandwiddi of the link connecting die source connectivity between two points in a TCP/IP network (i.e^ 

to the destination or to a value rqvesentative of a rate less in determining if two points are actually connected). In 

than the available bandwidth. That is, if the available band- 20 TCP/IP. a ping packa is sent to the desired end point which 

width is 1 Mbps, the rate parameter can be set to any value reverses the addresses and sends it back to die sender. The 

between zero and one-thousand, where 1000 Kbps equals 1 roundtry> time delay is also measured, and this is a mea- 

Mbps. This ability to explicitly set the transfer rate allows surcment of the time it takes foo* die ptng packet to travel 

long (in time) file transfers to coexist with other applications fixxm the sender to the desired end point and then back to the 

on the network widiout hogging all or substantially all of die sender 

bandwidth of die network. Jt is also desired to provide a multicast piiig utility, where 

MULTICAST GROUPS all the mcml>ers of a multicast group respond to the ping 

**Multicasr is defined hereinabove as the case when die packet or ping request Clients or hosts that support multi- 
server node 20 sends data (e.g., a file) to a subset of all <tf ^ cast IP (RFCim) will respond to a ping request widi a 
die dient nodes 22 connected to die network 24. It is also Class DIP address as the destination address. However, in 
disclosed hereinabove that multicast transmBsion can be in known multicast in^ilementations, die sender of the ping 
two forms: **application layer (AL) multicasT and *imilti- request only displays die firstresponse it receives to its ping 
cast IF^. AL multicast is used when die network does not request That is, known multicast ping techniques do not 
support the imemet specificaticm RFC1112 but does support 35 make a netwcvk connectivity measurement 
broadcast if nmlticast IP is suppoirted by die netwMk A *^ulticast Ping- feature of die invention dismays all 
according to RFC1112 and multicast IP routing, it is rco- multicast responses to a jnng request diereby providing die 
ommended over AL Multicast Multicast IP is used when netwoxk connectivity information from source to group 
members of the group must support multicast and routers in ^ recipients, and the roundtrip time delay information for each 
die router network must also support some kind of multicast multicast ffoop rec^ent In a pcefeired embodiment, diis 
roiling protocol (e.g., DVMRP, MOSI^, or FIM)l Unlike feature uses die standard ping K:>IP messages. 
AL multicast. Multicast IP is a true siulticast protocol where As an enhancement acccniing to die invention, it is also 
only mcmbcrs of die nniltica^ group receive die transmitted possible to use die Announcc/Registr^on fedlity (fescribed 
<tetft- 45 hereinbefcffc as aiiodier form ofdie**MulticastPingr feature. 

For each file transfer, a multicast ^oup can be defined Widi this enhancement Announce/Registration ping mes- 

duiing die ANNOUNCEmECHCTRAnON aspca of data sages determine connectivity to die application layer of die 

transmission, as describe hereinbefore. As stated, the server group recipients and back to die sender and roundtr^ time 

maintains various information about the file transfer and die delay infcrmatiott for eadi group recipient 

participants ot group involved in die transfer. In die pre- ^ The *Ti4ulticast Ping** feature dius aUows network con- 

fexred embodliment this infonnation is maintained by die nectivity and roundtr^ delays to be determiDed tyy the 

server in die form of data structures or lists. The server sender for membcis of a multic^ group, 

maintains and u ses diis information to record and determine MUITICACT NETWORK PROBE 

the stams of the file transfer during die DATA TRANSFER 55 Multicast (the sending of one to many, but not to all) dau 

stage: The client statos structure includes a list of the ^atns netwoiks are just now starting to be in^nnented. Multicast 

of the participants of the multicast groq> based on data from DP, in particular, is new in router netwcsics and can provide 

the announce registrations that are received by die server. the mcdiam'sm for creating multicast groups over networks 

Multicast group management is die process of assigning of all kinds. e.g.. frame relay. SMDS, LANs, satellite 

clients to multicast groups. The task of organizing and ^ wireless. The Internet also has die "Mbone"* (multicast 

manipulating the list of clients In each group is the lespon- backboneX a part of die Internet diat supports nmlticast IP. 

sibilfty of die application program that initiates file transfer The Mbone was started in early 1992 and has grown so 

in the first instance. The application program generally that at the beginning oi 1995 over 1500 subnets of the 

provides ease-of-use features such as associating a name ^ Intemet were multicast enabled. To this point, the Mbone 

widi a diem IP address, assigning a name to a group, etc has been used as an experimental network by Internet 

Groi^ mana ge m e nt is required only at the sendii^ station, researchers who have tested PC and workstation based video 
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conferencing and whiteboard nmtticast ^;>plicatioiLS, as well 
as Internee '*radio** and odier experimental ^»plications. 
Multicast IP routing on Ihe Mbone was initiaUy imple- 
xnented in wc^tations using die multicast routing protDco} 
DVMRP; however, parts of the Mbone have had tbdr 
routers upgraded so they are multicast enabled. It is antid- 
pated that within 5 to 6 years the Internet will be fully 
multicast en^ted using the routers in the Internet 

As more of die Internet becomes multicast enabled it will 
be used for mainstream multicast applications rather than as 
an experimental researdi tool. As this occurs, tools will be 
needed to facilitate usage. 
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and protocol described herein. After Announce^gistration 
to a multicast group of members A through E is used as a 
means to determine connectivity (Le.. to determine which 
members are actually connected to the server) in accordance 
with, fw example, die ''Multicast Ping" feature of the 
invention described in the preceding section, a test suite of 
small files are sent in sequence at different speeds to tiie 
group members. For example, a 400-fraroe test file may be 
sent first at 64 Kbps. then 128 Kbps. dien 256 Kbps. then 
512 Kbps. and finally at 1024 Kbps. Client negative 
acknowledgements will be received and stored at the server 
as shown in TaUe 1 below, assuming no other traffic on the 



TABLE 1 



Test Rcaohs wiA ■400-&qae Test File 
S|)eedSeiit #NakftforA fNib&rB tfNaksfarC ffNaksfbrD #Na]EsforE 

0 
0 
0 
0 
0 



64 Kbps 


0 


0 


0 


0 


128 Kbps 


200 


0 


0 


0 


256 Kbps 


300 


200 


0 


0 


M2Kbio 


350 


300 


200 


0 


1024 Kbps 


375 


350 


300 


2CX} 



One laige difference between the Internet and a private 
network is that the Internet is a vety heterogeneous network. 
It is a netwoik of networks, and there arc large differences 
in the different parts of the netwoik opoatcd by different 
Gcganizations. In contrast* many pnysA& networks are s^ up 
to be relatively h<»n0geneous, with much control by die 
operator of the private network as to the aidutectme of the 
netwwk- 

Since the en(]^»o>tnts in the multicast netwok are likely to 
be linked at different rates with different networks, and 
conge^ion in the network will be difiiercnt at different parts 
of the network, it is desirable to be able to gain knowledge 
of the capacity of die attached links in the muldcast group, 
and to test pecfoimance at that capacity. A "T^ulticast 
Network Probe** feature of the invention is designed to be 
able to probe die Mbcme or other large heterogeneous 
multicast necwoA from the trafiBc source and measure the 
capacity of the individual links quickly from that trafiSc 
source. 

Referring to FIG. 6, a heterogeneous multicast network 
(e.g., the Mbone portion of the Internet) has a multicast 
group widi five monbers, A dntMig^ E^ wtae each member 
of die group is oonnected by a diffierent cqkadty lii^ Le.. a 
different rate link. Member A of the group is tied to the 
network with a 64 Kbps (kilobits per second) link. B with a 
128 Kbps link. C widi a 256 Kbps link. D widi a 5 12 Kbps 
link, and E with a 1024 Kbps littk. The nature of ttiese link 
connections is unknown to the server (Le...die traffic souxce) 
because connections to the Internet can be at many different 
speed links. 

It is desirable fot the traffic source to know the charac- 
teristics of the links to destinations so that it can optimally 
determine how to perform the multicast transfer of infor- 
mation to the desdnadons. die q)plication is a video 
conference, it may be determined diat die quality to A at 64 
Kbps may be unacceptable, but the rest could participate at 
12S Kbps. Similarly, if file transfa is die application, groups 
D and E could make up a group operating at 512 Kbps 
transfer rate, while groups A. B, and C could operate at 64 
Kbps without exceeding network capacity. 

In accordance widi the invendoa. die mechanism to probe 
the networic to determine remote link capacity is the system 



Refecting to Table 1, the first run at a ^lecd of 64 Kbps 
results in no negative acknowledgements (Lc.. NAKs or 
Naks) for any of the group members because all links 

^ support 64 Kbps or greater. 

The second run is at 128 Kbps. twice that of die first In 
diis second run, dient A has 200 NAKs, meaning that half 
the feunes are lost This means diat the speed of client A is 
64 Kl^ (i.e., ((40Q-200)/400)*128 Kbps==64 Kbps). C3ients 
B through E exhibit no franie loss in the second run. and thus 
die speed of each of those dieots is at least 128 Kbps. 

In the third run. the speed of transfer Is 256 Kbps. and 
clients A and B exhibit 300 and 50 lost frames, respectively. 
Thus, from this third run. client A's speed is 64 Kbps (Le., 

^ ((400-300y400)*256 Kbps=64 Kbps) which confirms die 
measurement firom the second tun. Also, in the third run. 
client B*s speed is 128 Kbps (Le., {(400-200)/400)*256 
Kbps=:128 Kbps). Qients C through Bhave no cnors in this 
third run, and thus diey eadi c^oate at least as fast as 256 

45 Kbps. 

In tbsi fourth nut the speed of transfer is 5 12 Kbps. Cli^t 
A exhibits 350 lost frames so measures (400-350)^400*5 12 
Kbps or 64 Kbps which diecks with die previous measure- 
ments. Chsat B exhibits 300 lost frames which measures 

50 ((400-300)/400>*512 Kbps or 128 Kbps which also checks 
witti previous runs. Client C exhiHts 200 lost firames which 
measures ((40a-200y400)*512 Kbps or 256 Kbps. 
In the fifth run, die speed of transfer is 1024 Kbps. Client 

. A exhibits 375 lost frames which measures to ((400-375)/ 

„ 400)*1024 or 64 Kbps as before. Client B measures 
((400-350V400)*1024 or 128 Kbps, and client C measures 
((400-300)/400)*1024 or 256 Kbps. Client D measures 
((40O-200)/400)*1024 or 512 Kbps. dient E has no drops 
which means diat its speed is at least 1024 Kbps. 

^ Thus, for eadi of die five runs, the capacity of a given link 
is given by the following equation: 

({ffrvwt 3aic-4teaksyiframcs 8eiit>*8peed of tnosDoissioi^link 

63 

This test technique also will take into account the traffic 
on the Hnk. For exanqile, if aphysical link is 256 Kbps and 
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there is 128 Kbps of traffic on ftc link when the test is NEGATIVE ACKNOWLEDGEMENT CCttXECTION 

pexf(»3iied, the measurement wiU come up with a cq>adtyc^ As mentioned previously hereinbefc^e, the number of 

128 Kbps. the remaining edacity when the traffic is con- clients which can receive a file according to the invention 

sidered. ^ number in the thousands. Thus, die zmmber of entries in 

Software for in^lementiog these tests can also be used to 5 client status list maintained by the server can number in 

test the quaUty of links given tiiat the source knows the Hnk thousands. File transfer according to the invention can be 

speeds to each cUent. For example, in FIG. 6, the link speeds "^o^e scalable. For example, it can be scaled to send 

may be known and it is desired to test the links with afiletomillionsofrec^>ients/dientsinsteadof thousands of 

leUtivcly long test patterns to determine foimc crit^r rales. ^^cipicnts/cUents. In a prcf OTed embodiment, thwe clients 

For example, a 100,00(>.&ame test file could be sent at 64 lo ?^^!f 

Kbps to Ite group consisting of members A through R Hie TT>e scaling feature ^ helpfiil to avoid a potential problem 

^ rl" ^. " jTT *j. ^Yica the nunibcr of dtents in the group become too large, 

rate of tmn^sion and the NAIfe ai>e s^^^ at the source. ^^^^ ^ ^ P ^^^^^ ^^^^^^ 

and the niimb<^ of N^O^ from each cUent gives a ineasure ^^^^ acknowledgemeots to the file sendff (e.g., server) 
ofthequakty(i.e..thefirameeirar^^^ and effectively choke the sender with more negative 
be expected tiiat A would have tiie worse quahty as It is Ac 15 acknowledgements than it can handle In a reasonable period 
most heavily loaded link, and E would be best as it is tfic of time. This causes the performance of the sender to drop 
least loaded. However, other factors could cause other as it needs to spend a significant amount of time receiving 
results. Similarly, speeds can be increased and overloaded and processing the negative acknowledgements and it can- 
links may be deleted from the group to m<»e heavily stress not attend to its other duties. This also clogs the link hack to 
the higher speed links. 20 the sender to become jammed with the traffic of diese 

Thus, using the *^ulticast Network nrobe"* feature of die negative acknowledgements, 
invention, the opacity of individual links can be measured The solution to the problem is "native acknowledge- 
quickly by the server if the individual link capacities are nicnt coUection** whidi in turn allows the number of client 
unknown. Alsa if the link speeds are known by the server. recipients to be dramatically increased firom thousands to 
this feature of the invention can be used to determine the 2S nullions without clogging at the file soider/server 20. With 
quality of each link (i.e., to determine the frame error rate of ^ collection feature, certain clients or other network nodes 
each link). ^ ^ **replication points** and collect block negative 

In accordance with this feature of the invention, the acknowledgements from other clients. In a preferred 

connectivity of the mendxirs ol the mniticast gfovp is first embodiinent, these r^licadon points (RPs) are routers, 

determined by going through an ANNOUNCE/ 30 Itefaring to HG. 7. five RPis are indicated across the United 

REGISTRATION phase desGrit)ed hereiribefore. That is, the States, and the lines emanating from each RPare rcpresen- 

inifiai step is to determine which noenibers of the group are tative of the one or more cUents cormected to that RP. For 

connected to die server: Once the cormected members are exaxr^le, RP 100 has 1200 clients thereunder, RP 102 has 

known, the test file transfer can begin to dctamine Hnk 900 clients^ RP 104 has 100 clients. RP 106 has 800 clients. 

spGcd or quality by the server sending a test Me to each 53 ^ clients. The server or source 20 is 

menKbcr and recording the results (Ic^ the number of located at another place in the United States. RP 100 collects 

negative acknowledgements for each group member). all of the block negative admowledgements from the (e.g., 

SPEED CHIOUPS 1200) clients associated therewith or ooiuiected thereto. The 

With the knowledge <^ the capacity, speed, or bandwidd) ^^^^ do the same fr»^ dicir associated 

of eadi of the various Kninc interfacing the saver to the 40 clients. For each RP. after it collects all block negative 

clients (made availaMe by, for cxanqide^ the *14ulticast acknowledgements from all ofits associated diects, that RP 

Network Probe** feature described in the preoedmg section), sends (m to die .server 20, or to anodter RP in the chain 

alist of diese speeds can be stored by the server. The list can heading to the server 20, just one acknowledgement mes- 

dicnbeusedtogcnciateordcfineapluralityofcHentgroups message indudes all of the block n^ative 

based on Knir speed. For exanq>le. there may be two speed 45 acknowledgenieiits from all of the clients associated with 

gnxips where one indudes a client connected to the server When the server 20 eventually receives diesc 

over a link (or effective link) having a mflximiim possible collected block negative acknowdcdgement messages frtxn 

speed 64 Kbps and where the other one indudes a dient the RPs, it sends back out on the next pass all of the frames 

connected to the server over a link (or effective link) having negatively acknowledged. The RPs are responsible for 

amaximumpossiblespeedof 1024 Kbps. The second group so receiving those subsequent-pass frames and forwarding 

is thus much faster than the iirst group. What ^>eed ^oup a ^ q)propiiate clients <v other RP in die chain 

particular recqnent is in affects the transfer of data to that which will then forward them to the ^propriate dients or 

redpient During the initial pass of data transfer according RP in the diain. etc. 

to the invention, each of the recqMents in tht second, faster Variations, modi fi ca t ions, and other in^>lemcntations of 

speed group will be sent all the frames by the server, and 55 whatisdescribedhcretnwilloocurtothoseofordiiiary skill 

each of die rec^ients in the first, slower group wiU be sent in the art without departing from die spirit and the scope of 

only every sixteradi frame (V^s) sent to die second group. invention as daimed. Accordingly, die invention is to t>e 

This means that after die first pass, the server has sent all defined not by die preceding illustrative description but 

frames to the second group rec^ents, but it has only »mt instead by die following claims. 

one-sixtcendK^ttietotalnumbcrofframestothefirstgroiq). 60 What is daimed is: 

The remaining postkm of the firames not yet sent to the first 1- ^ method for transmitting data over a comnumications 

^oup (i.e., of the frames) are then sent to the first group conqnising: 

rec^iients on sut>sequent passes. The point being that once (A) setting a maximum data transmission rate to a value 

the server knows the capacity of each member of a group, less than or equal to an available bandwidth of the 

die server can tailor the data transfer to take advantage of the 65 communications link; 

higher capadiy links and not slow ckiwn the transfer of data (B) partitioning the data into a plurality cf blocks whidi 

thereto. each indudes a plurality of frames; 
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(C) transmitting all of the frames to one or more rec^i- 
ents; 

(D) during transnussion, receiving acknowledgments 
from the recq[>ient5 wtdch include indications of frames 
requiting retransmission; and 

(E) repeating steps (C), (D), and (E) for only ttiosc tiroes 
which the acknowledgments indicate require retrans- 
mission. 

X A method for quiddy and reliahly transmitting data to 
at least two rec^ents over a oomnuinicattons linku com- 
prising: 

(A) setting a maximum data transmissioQ rate to a value 
less than or equal to an available bandwidth of the 
commuiucations link; 

(B) transmitting a phirality of frames of data over the link 
to die recipients untU all of the plurality of frames have 
been transmitted; 

(Q while pcrfonning step (B)« receiving acknowledg- 
ments from one or mere of fte reci]»ent$. the admovtl- 
edgments including indications of frames requiring 
retransmission; and 

(D) after all of the jdurality of frames have been 
transmitted. rq>eating steps (B)<, (C>, and (D) for only 
those frames which the acknowledgments indicate 
require retransmission. 

3. The method of claim 2 wherein steps (B)« (Q. and (D) 
are repeated, as recited in st^ (DX until no frames require 
letransmission. 

4. Hie method of daim 2 wlicran stq>s (B), (C). and (D) 
are repeated, as rcdted in slq;> (D). untQ a predetermined 
amount of time has passed. 

5. A method for transmitting data over a communications 
link, comprising: 

(A) defining a multicast group of rec^ients to receive the 
data v^KJDcin the group inciudrs a subset of all recipi- 
ents; 

(B) paititi<Hung the data into a plurality of blocks which 
each includes a plurality of frames; 

(C) transmitting all (tf the frames to the multicast group; 

(D) during transmission^ receiving acknowledgments 
from the recQttents in ^ multicast group, die acknowl- 
edgements inchtding indicstioas of frames requiring 
retransmissicm; and 

(E) repeating steps (C), (D), and (E) for only those frames 
which the admowled^;ments Indicate require letrans- 
mtssion. 

6. A method for quickly and reliably transmitting data to 
a multicast group of recqnents over a communications link. 

^ipip pfri SI ng ! 

(A) defining the multicast group of redpicots to receive 
the data wherein the group includes a subset <^ all 
recipients; 

(B) transmitting a jduraiity of frames of data over the link 
to the multicast groiq> until all of the plurality of frames 
have been trananittrd; 

(Q wbSc pcrfonning step (B). receiving acknowledge 
ments from one or more <^ the rec^ients in the mul- 
ticast groiq>. die acknowledgments including indica- 
tions frames requiring retransmission; and 

(D) after all of the plurality of frames have been 
transmitted. rq)eating 5tq>8 (B), (C)« and (D) for only 
those frames whidi the acknowtedgmeots indicate 
require retransmlssioa. 

T.Hie methodof daun6 whereiD steps (B). (C). and (D> 
are repeated, as rcdted in step (D). until no frames require 
retransmission. 
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8. The method of claim 6 wherein steps (B). (C). and (D) 
are repeated, as recited in step (D). until a predetermitted 
amount of time has passed. 

9. The mediod of daim 6 fimher comprising, prior to step 
(A)« sending a ping request to all rodpients and recdving 
responses £rom the recqnents that are connected to a source 
of the ping request In order to determine which redpients are 
available to be in the multicast group. 

10. A method for dmnnining the capacities of commu- 
nicatimi Imfcs cotinecting rec;Q)ients to a source, comprising. 

(A) detcnnining which recqncnts are omnected to the 
source by the communicatioD links, each redpient 
being ootuected to die source by a dii£ferent one of the 
oommunication links; 

(B) transmitting a plurality of frames of data from the 
source to the redpients determined in step (A) at a 
pcedetermined rate over the oommunication links until 
all of the i^urality of frames have been transmitted; 

(O vAdle peifonning step (B), recdving acknowledg- 
ments from the redj^ents dctcmiined in step (A), the 
acknowledgements including indications of frames 
requiting retransmission; 

(D) storing the acteowledgements and the predetermined 
rate at the source; 

(E) rating steps (B), (Q, (D), and (E) for a different 
pcedetermined rate until steps (B). (C). (D), and (E) 
have been repeated a predetermined number of times; 
and 

(F) determining oq^acity of one or more d the commu- 
nication links from information stored at the source. 

11. The method of claim IQ wherein step (A) is pcrfonned 
by the source sending a ping request to the rec^lents over 
tiie commamcatioa links and the source receivii^ responses 
from the recq»ients that are connected to the source, and 
wherein the redpients are menolm of a multicast group. 

12. Hie method of daim 19 further comprising, ahcr step 
(F): 

{G) transmitting a plurality of frames of odicr data from 
the source to at least one of the rec^ents which is 
connected to the source by one of the communication 
links determined in step (I^ to have a first capadty; and 

(H) transmitting a subset of the plurality cf frames of 
odier data firom the source to at least one other of the 
rec^ents which is connected to the source by one of 
the oomniunication links detetmined in step (F) to have 
a second o^Nuity where the first capacity is higher than 
the second cs^Mctty. 

13. A method for determining Ihe firame errcr rates of 
communicadon Unks connecting rec^ttents to a source, 
oompnsing: 

(A) determining w\nd\ recqHents are connected to the 
source by the commiihication links, eadi redpient 
being connected to the source by a different one of the 
comsuinicatioa ,links; 

(B) transmitting a plurality of frames data from the 
source to the reelects detcnnined in slq> (A) at a 
predctcrndned rate over the communication links until 
all of die plurality <^ frames have been transmitted; 

(O while pcrfonning step (B)* receiving acknowledg- 
ments from the redpients determined in step (A), the 
acknowledgements including indications of frames 
requiring retransmissicMi; 

(D) storing the ackDowtedgemeots and the predetennined 
rate at die source; and 

(E) determining frame error rate of one or more of the 
communication links from infccotation stored at die 
source. 
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14. The method of claim 13 wherein step (A) is pcff oxmcd 
by the source sending a ping request to the recipients over 
the communication links and the source receiving responses 
from the rec^ients that arc oonoected to the s<MBce, and 
wb^ein the recipients are members of a multicast group. 

15. A method for determining the connectivity between a 
source and members of a multicast group on a network, 
comprising: 

sending a ping request from the source over the network 

to all of the members of the multicast group; 
receiving at die source responses to the ping request from 

all of die members of the multicast group; and 
determining, at the source, roundtrip delay for the ping 

request to travel to each of the members of the multicast 

group and back to the source. 

16. A nffithod for transmitting data, comprising: 

(A) transmitting a plurality of frames of data frx>m a 
source to at least one recipient which is connected to 
the source by a iirst communication link having a first 
capacity; 

(B) transmitting a subset of the plurality of frames <^ the 
data from the source to at least one other recqnent 
which is connected to die source by a second commu- 
nication link having a second capacity where the first 
capacity is higher than the second a^>acity; 

(Q transmitting die plurality of frames ova the first link 
and the subset of the plurality frames over the second 
link until all of the plurality of frames have been 
transmitted over the first link; 
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(D) while poforming step (C). receiving acknowledg- 
ments from the recipients connected to the source by 
the first and seamd links, the acknowledgments includ- 
ing indications of frames requiring retransmission; and 
5 (E) after all of the plurality cf frames have been trans- 
mitted over the first link repeating st^ (C). (D). and 
(E) for only those frames whidi the acknowledgments 
Indicate require retransmission. 

17. A mediod for quickly and reliably transmitting data to 
10 a laige number of recipients, oon^sing: 

(A) transmitting a plurality of frames of data over a link 
through a replication point to the recipients until all of 
the plurality of frames have been transmitted; 

(B) \^e peifonning step (B). receiving and collecting 
15 acknowledgments from all of the recipients at the 

replication point, the acknowledgments including indi- 
cations frames requiring retransmission; 

(C) passing on the received and collected acknowledge- 
ments from the replication point as an indlcadon of 

20 frames requiring retransmission for all of the recipients; 

(D) after all of die plurality of fr^es have been 
transmitted, lepeatii^ stq>s (A). (B). (C). and (D) for 
only diosc frames which the replication point has 
indicated require retransmission. 

23 18. The mediod of claim 17 wbmin steps (A). (B), (Q, 
and (D) are repeated, as recited in step (D), until no frames 
require retransmission, and wherein the r^lication point 
comprises a router. 

♦ * ♦ ♦ » 
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